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Introduction 


Over the last few years GMAO has been developing the tangent 
linear and adjoint of GEOS-5. 


This system supports a number of current and upcoming 
operational and research capabilities: 


Daily calculation of observation impacts (FSOI) 
4DVAR data assimilation using GSI 

Adjoint sensitivity to initial conditions 

Singular vector calculations 

Assessment of new observations and changes to current 
Estimation of surface fluxes of emissions 


Analyze ensemble techniques 


Introduction 


We have a nonlinear model for predicting the future atmosphere, 
yirm (x1, X2, gee ’ 
where x and y are discrete model variables at an old and new time. 


From a predictability perspective we may be interested in: 

1. How does a forecast change with respect to the inputs? If 
there’s some error or change due to an observation, how does 
that quantity grow? 

2. What caused some aspect of the forecast we're analyzing, e.g. 
an error? What specifically was it about the initial state that 
led to this? What was the value of an observation to the 
forecast? 


Introduction 


We're trying to get a handle on perturbations so we ‘linearize’ 
model variables by separating into reference and perturbation parts, 


= xl +4! 
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Introduction 


We're trying to get a handle on perturbations so we ‘linearize’ 
model variables by separating into reference and perturbation parts, 


= xf + x! 
see a 


The output perturbation can be approximated by Taylor expansion, 


y; =m Of + x}, x5 + x5, | = M04 9+); 


The above equation is the tangent linear model (TLM) and 
gives an approximation for the growth of perturbations xi. 


Introduction 


Introduction 


The TLM is a useful tool but doesn’t help us with the second 
question. Introduce a scalar measure of the outputs, J = J [y (x)]. 


Using the Taylor series again, 


oJ" oJ 
J= 5% or a) Dye 


The two estimates are equivalent when both functions are linear. 
J is a function of a function so expand with chain rule, 


As" (dy)\" Os" 
Oxy 2d (x) Oyj 


i 


Obtain a powerful equation from end time to beginning time. 


Introduction 


The TLM sums Oy;/0x; over j while this new model sums over /. 
Adopting vector notation to compute all y/ or 0J/Ox;, 


For the TLM sum over the columns, for this new model sum over 
rows. In standard matrix multiplication we sum over columns. The 
TLM is fine, 

y’ = Mx’, 


but for the new model we have to transpose (adjoint) the matrix, 
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Outline 


e Development of the GEOS-5 adjoint 
e Applications 


e Ongoing & future work 


Overview of the system 


The GEOS-5 adjoint consists of: 


e Finite volume cubed sphere (FV3) dynamical core 
e Relaxed-Arakawa convection 

e Bacmeister single moment cloud scheme 

e Turbulence based on Lock et al closure 

e Gravity wave drag 

e Long- and short-wave radiation 

e GOCART dust physics 


FV3 dynamical core 


The FV3 dynamical core is developed by NOAA GFDL and has 
been central to multiple versions of GEOS. 

e Finite volume numerics 

e Cubed-sphere grid 

e D-grid shallow water dynamics 

e Lagrangian vertical coordinate with PPM remapping to o-p 
e PPM horizontal advection schemes (around 15 options) 
Through NGGPS saw the following updates, 

e Updated non-hydrostatic algorithm 

e Single precision ( 30% faster) 

e Nested grid capability 


FV3 dynamical core 
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Pressure (hPa) 


Strong versus weak nonlinearity 


Dynamical cores largely exhibit only ‘weak’ nonlinearity. By this we 
mean that the following condition holds, 


lim m(x + Ax) — m(x) = MAx 


Ax->0 


Physics schemes exhibit ‘strong’ nonlinearity. Due to the use of 
piecewise (linear or otherwise) functions, 


lim _m(x + Ax) — m(x) 4 MAx 
Ax-—0 


In earlier NWP models it was possible to have correlations of 
almost 1 between tangent linear and nonlinear dynamical cores out 
to 5 days or so. 


Strong nonlinearity in dynamical cores 


The primitive equations of the atmosphere are weakly nonlinear. 


However, these equations are not solved in their continuous form 
but discretised and solved numerically, with a finite volume method 
in FV3's case. In doing so we run into problems, e.g. linear 
positive definite advection schemes cannot be greater than first 
order (Godunov, 1959). 


As a result sophisticated nonlinear shape preserving schemes are 
employed to solve the horizontal and vertical (remapping) 
transport. 


Strong nonlinearity in dynamical cores 
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Strong nonlinearity in dynamical cores 
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Strong nonlinearity in dynamical cores 


These strong nonlinearities generally also produce linear instability 
and thus uncontrolled growth of the perturbations. 
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(c) Time series of maximum Re(A) 


A 2D spherical geometry test case using prescribed winds and true 
solutions has been developed to test the approach to linearized 
advection in NWP (Kent and Holdaway, 2017, QJRMS). 


Transport in the FV3 adjoint 


All positive definite PPM schemes available in FV3 are found to 
exhibit poor tangent linear behavior. Instead transport is handled 
by purely linear differentiation (weakly nonlinear overall due to 
varying winds). 


A third order method is introduced to provide a scheme with more 
diffusion than the fifth order linear scheme present in FV3. In the 
sponge layer first order advection is used. 


For remapping a first order linear interpolation is implemented for 
all perturbation quantities. Cubic was also tested but not found to 
produce good results. 


Transport in the FV3 adjoint 


5 day integration of a slotted cylinder tracer perturbation with 
wind perturbations set to zero. Third order linear differentiation for 
the tracer and linear remapping. 
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May be beneficial to add some targeted vertical hyper-diffusion to 
the tracer perturbations. 


Damping in the FV3 adjoint 


Divergence, vorticity and external mode damping in FV3 are tuned 
in concert with the transport scheme. Some schemes are more 
diffusive than others, requiring less damping to maintain stability. 


For the linearized version of FV3 significantly more damping is 
required to maintain stability. Typically we need to run with 
second and fourth order divergence damping coefficients maxed 
out and with voracity damping set quite high. 


Reference - perturbation splitting 


We have three choices when running the adjoint: 


e Run both reference and perturbations with the transport and 
damping schemes that work well for the perturbations. 

e Split the transport so reference values use the nonlinear schemes 
and perturbations use the linear schemes but both use the 
divergence damping that works for the perturbations. 


e Split both the transport and damping schemes. 
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Splitting is slower but more accurate, useful for longer runs looking 
at sensitivity or FSOI. For 4DVAR and singular vector calculations 
it may be beneficial to turn splitting off. 


Testing: Jablonowski-Williamson Baroclinic Instability 
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Testing: Analysis Increment Perturbation 


Correlation between the 24 hour nonlinear and tangent linear 
perturbation trajectories. Blue = old TLM, red = new TLM. 
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uv = 0.81, v = 0.81, T, = 0.79, g' = 0.51, Ap’ = 0.92. 


Testing: Analysis Increment Perturbation 


Root mean squared error between the 24 hour nonlinear and 
tangent linear perturbation trajectories. 


Model Level 
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Testing: Analysis Increment Perturbation 


Root mean square compared to nonlinear model. 
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Testing: Analysis Increment Perturbation 


100hPa nonlinear and tangent linear model perturbation fields. 
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Autodifferentiation 


The Tapenade tool is used to generate the majority of the code. 
Pros: 


e Very flexible and works on modular & object form of FV3 
e Fast, generates FV3 TLM and adjoint in matter of minutes 
e Minimal re-computation (fast) 

e Fast writing of checkpoints to stack using malloc() 

e Inexpensive 

Cons: 


e Minimal re-computation (memory intensive, or how to bring 
down a node in one easy step!) 


e Precision determined at generation time 


Timing 


The adjoint of FV3 is about 4 times slower than the nonlinear 
model, the tangent linear is around 2 times slower. 


Adjoint Model 


‘Tangent Linear Model| 
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FV3 has a relatively small timestep, 75s when Ax = 50km. On the 
other hand it resolves small scale features very well, one the 
reasons it was picked in the NGGPS testing. 


The only application for which this is a major concern is 4DVAR, 
but there are things we might be able to do. 


Autodifferentiation 


We have written our own push/pop checkpointing routines, 
allowing for compile time precision choice. This has other benefits: 
e Flexibility of memory handling, malloc/SHMEM /allocate. 

e Check on superfluous saves, especially around communication. 


e Ability to hold checkpoints in memory. This could save a 
tremendous amount of time in 4DVAR if we can spare the 
memory. 


Outline 


e Applications 


e Ongoing & future work 


Sudden Stratospheric Warming 
In 2002 a rare southern hemisphere sudden stratopsheric warming 
occurred, disrupting the polar vortex and temporarily eliminating 
the ozone hole. 
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Sudden Stratospheric Warming 


The adjoint model serves as a useful tool for examining the 
sensitivity to initial conditions ahead of the event. The method 
works as follows: 


Choose an initial metric J, here a small area at 10hPa where the 
warming first occurs. 


Run the adjoint backwards to find sensitivity to initial conditions 
2-3 days before. 


Generate a perturbation that minimizes J using a Lagrange 
multiple technique. 


Run a new nonlinear forecast with initial conditions altered by 
the optimal perturbation. 


Verify the perturbation. 


Examine how the perturbation evolves. 
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Sudden Stratospheric Warming 
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Sudden Stratospheric Warming 


5 day running mean of the vertical component of the Plumb wave 
flux centered 24 hours before the first significant warming occurs. 


Sudden Stratospheric Warming 


5 day running mean of the zonally asymmetrical height centered 
ahead of the warming. 


Zonally-asymmetric component of Height 
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Sudden Stratospheric Warming 


The sudden stratospheric warming is shown to be driven by Rossby 
waves emanating from tropospheric disturbances. 


This is further confirmed by examining the ERA-20C reanalysis. 
Here only surface observations are used to derive a century long 
atmospheric state that includes the stratosphere. The 2002 
southern hemisphere sudden stratospheric warming is evident in 
the record, though a little weaker than it was in reality. Further, 
there is evidence for 3 more southern hemisphere sudden 
stratospheric warmings, all occurring in the early 20th century. 


Hurricane Joaquin 


Hurricane Joaquin was a category 4 Atlantic hurricane in 2015 that 
models incorrectly forecast as making landfall on the Eastern 
seaboard. 


11L Tracks from 1200 UTC 29 SEP 2015 


Forecast Tracks for Hurricane-4 Joaquin 
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Hurricane Joaquin 


36 hour sensitivity of kinetic energy to the initial wind fields, 


COAMPS versus GEOS-5 adjont models. 
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Hurricane Joaquin 
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Hurricane Joaquin 
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Hurricane Joaquin 
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Hurricane Joaquin 


Accurate forecasting Joaquin was immensely difficult due to the 
high degree of sensitivity to the initial conditions. 


Small changes to the initial conditions results in changes to the 
intensity which changes the track significantly, leading to even 
greater changes in intensity. 


Sensitivity to Saharan dust in hurricane formation 


TC formation requires heat and moisture, which can be affected by 
dust in a number of ways: 


e Dust absorbs solar radiation and warms up relative to the 
surroundings. 


e However, this prevents some solar radiation reaching the surface, 
cooling below the dusty layer. 


e Dusty air tends to be dry and as the SAL is entrained it can 
reduce the available energy. 


e Dust can impact condensation through micro-physical processes. 


Perturbation vs. sensitivity evolution 


Evolution of temperature sensitivity evolves very differently than 
changes in temperature due to an optimal dust perturbation. 
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Optimal dust removal 
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e Helene (left) weakens and moves north. 
e Other storm (right) strengthens and moves south. 


FSOI 


GMAO is a partner in the international FSOI inter-comparison 
project with JMA, NRL, EMC, Meteo France and the Met Office. 
We have run two seasons of 4 times daily adjoint observation 
impacts. 


24-h Observation Impact Summary 
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Winter storm evolution (U of Wisconsin) 


Winter storms of 1993 (predictable) and 2000 (unpredictable). 


Winter storm evolution (U of Wisconsin) 


1993 storm of the century case 
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Winter storm evolution (U of Wisconsin) 


2000 surprise snowstorm case. 
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Outline 


e Development of the GEOS-5 adjoint 
e Applications 


e Ongoing & future work 


FV3 adjoint development 


Finish testing the non-hydrostatic version of the code. 


Generate adjoint of nesting, which requires different MPI 
communication. 

Develop system for maintaining the checkpoints through 4DVAR 
minimization. 


Increase efficiency. 


GEOSctm Adjoint Model 


GMAO have recently developed a chemistry transport version of 
GEOS-5. Having an adjoint of this mostly linear system will allow 
for long window estimation of sensitivity to emissions. 


x10" Dust — BIN1 | 07-Sep-2006 0300UTC —_ Dust - BIN] (sens) | 07-Sep-2006 0300UTC 
4 


x10 
40 
2 Ty ae ane ti ; 
1S B= 30 . f te 30 
3 3 
A ia Sy Zi? 
5 00 205 
05 
ol 10 10 Py 
(kgkg ") (keke) 
Dust — Source | 07-Sep-2006 0300UTC Dust — Source (sens) | 07-Sep-2006 0300UTC x10" 
40 a woo 
os 5 _ 5 oy: 
0.6, 4E 30 oe)? 
3 ad 31 Jo 
of 42 eI 
a 3 
5 20 20 4 al _5 
02 
-4 
0 10 10 


—20 -10 0 10 0 
Longitude (°) Longitude (°) 


Singular vector development 


We have recently ‘revived’ the singular vector system in GEOS-5 
and development work is ongoing. 


e Continue testing the singular vector system comparing to the 
literature 

e Remove current restriction of horizontally global norm at initial 
time 

e Add further norms useful for examining tropical cyclone 
predictability 


Singular vector research 


e Compare singular vector growth with ensemble spread used in 
4DEnVar. 


e Apply the singular vector tool to the SSW problem. 


e Examine the low predictability hurricane Matthew case and the 
interaction with Nicole. 


Singular vector research - hurricane Matthew 


12 hour dry leading singular vector growth, total energy norm. 


Singular vector, field: u’ 
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‘TLM evolved singular vector, field: uv 
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Nonlinear evolved singular vector, field: u! 
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Singular vector, field: v 
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Hybrid 4DVAR 


We have begun testing the latest FV3 based adjoint in a hybrid 
A4DVAR system and hope to start examining results soon. 


Later we plan to make the adjoint of FV3 available to the 
community through the NGGPS and JEDI where testing of 4DVAR 
can continue and be compared to the current 4DEnVar techniques 
used at NCEP. 


Efficiency gains to the adjoint will be key to its success in the 
A4DVAR framework. We are working with computer scientists to at 
GMAO to identify ways to improve performance. 


Weak constraint?! 


Conclusions 


GMAO have developed sophisticated tangent linear and adjoint 
versions of the GEOS-5 global NWP model. 


e Can be kept up to date with changes to FV3 using Tapenade 
approach. 

e The system is scientifically sound but needs to be sped up. 

e Lots of interesting research questions to address, especially with 
the nesting. 


